Articulated model registration apparatus and method

ABSTRACT

The present invention relates to an articulated model registration method and apparatus which three-dimensionally restore an articulated model of a body, such as a hand, an upper body, and a whole body, from an image and to acquire a two-dimensional landmark from a two-dimensional image to generate and estimate a three-dimensional articulated model corresponding thereto.

TECHNICAL FIELD

The present invention relates to the three-dimensionally restoration ofan articulated model of a body, such as a hand, an upper body, and awhole body, from an image and to the acquisition of a two-dimensionallandmark from a two-dimensional image to generate and estimate athree-dimensional articulated model corresponding thereto.

BACKGROUND ART

A technology which three-dimensionally restores an articulated model ofa body, for example, a hand, an upper body, or a whole body, from animage, has been continuously studied before potable photographing anddisplay devices such as a smart phone become popular in recent years.Further, there are existing studies using a three-dimensional input suchas an RGBD camera which acquires depth information as well as atwo-dimensional image and a stereo camera which acquiresthree-dimensional information using a plurality of optical inputs.

Recently, as a result of rapid progress of a processor, studies whichthree-dimensionally restore an articulated model of a hand or a body byutilizing machine learning using a fast computing ability of aprocessor, especially, a deep learning technique are being performed.

Among such articulated model restoring techniques, there is a techniquewhich three-dimensionally restores the articulated model using aninverse kinematics (IK) technique after acquiring a two-dimensionallandmark using a deep learning technique. However, according to thistechnique, since a cost function is optimized using a Jacobian matrix ora conjugate gradient method, the computation is complicated so that along processing time is required and specifically, power consumption ofthe portable device is excessive and there is a risk to fall into alocal minimum.

DISCLOSURE Technical Problem

The present invention relates to an articulated model registrationapparatus and method which three-dimensionally restore an articulatedmodel of a body from an image and acquire a two-dimensional landmarkfrom a two-dimensional image to estimate a three-dimensional articulatedmodel corresponding thereto with a less computation amount so that lessprocessing time is consumed, a local minimum problem is solved and powerconsumption of a portable device such as a smart phone is reduced.

Technical Solution

The present invention provides an articulated model registration methodincluding: a body image detecting step of detecting an image of a bodyarea from an original image, a landmark extracting step of extractingone or more landmarks including an articulated point from the detectedimage of the body area, a body shape model generating step of generatingan average three-dimensional body shape model, a conversion estimatingstep of estimating a position on a three-dimensional coordinate of thegenerated body shape model from a conversion estimating landmarkincluding any one or more landmarks among the extracted landmarks, and aposture estimating step of estimating a position on a three-dimensionalcoordinate of the generated body shape model from any one or morelandmarks among the extracted landmarks in which the posture estimatingunit estimates the position on the three-dimensional coordinate of thelandmark from a reference straight line connecting the position on thethree-dimensional coordinate estimated from any one or more of thelandmarks, in an arbitrary position on the three-dimensional coordinate.

Further, the present invention provides an articulated modelregistration apparatus, including: a body image detecting unit whichdetects an image of a body area from an original image, a landmarkextracting unit which extracts one or more landmarks including anarticulated point from the detected image of the body area, a body shapemodel generating unit which generates an average three-dimensional bodyshape model, a conversion estimating unit which estimates a position ona three-dimensional coordinate of the generated body shape model fromone or more landmarks other than the articulated point among thelandmarks, and a posture estimating unit which estimates a position on athree-dimensional coordinate of the generated body shape model from oneor more articulated points among the landmarks in which the postureestimating unit estimates the position on the three-dimensionalcoordinate of the articulated point from a reference straight lineconnecting the position on the three-dimensional coordinate estimatedfrom any one or more of the articulated points, in an arbitrary positionon the three-dimensional coordinate.

Advantageous Effects

The articulated model registration apparatus and method according to thepresent invention acquire a two-dimensional landmark from atwo-dimensional image to estimate a three-dimensional articulated modelof a corresponding body, for example, a hand, an upper body, or a wholebody, with a less computation amount, so that the processing time isreduced by up to one-two-hundredth of the IK technique, andspecifically, the power consumption of the portable device such as asmart phone is reduced. Further, several candidate solutions arecalculated as a closed-form solution so that local minimum problemcaused when an optimization technique such as the IK technique is useddoes not occur.

DESCRIPTION OF DRAWINGS

FIG. 1 illustrates an exemplary embodiment of an articulated modelregistration apparatus 10 according to the present invention.

FIGS. 2 and 3 illustrate an exemplary embodiment of an articulated modelregistration apparatus 10 according to the present invention.

FIG. 4 illustrates an exemplary embodiment of an articulated modelregistration method according to the present invention.

FIG. 5 two-dimensionally illustrates a process for a step of calculatinga reference straight line.

FIG. 6 illustrates specific steps included in a posture estimating stepS15 according to the present invention.

FIG. 7 illustrates an example which estimates positions on athree-dimensional coordinate of a landmark P₀ corresponding to ashoulder of the body, a landmark P₁ corresponding to an elbow, and alandmark P₂ corresponding to a wrist of the end of the body in thisorder, by an iterative estimating step S153 according to the presentinvention.

FIG. 8 illustrates an example which estimates positions on athree-dimensional coordinate of a landmark P₀ corresponding to a rootjoint of the hand of the body, specifically, the index finger, landmarksP₁ and P₂ corresponding to two joints of the index finger, and alandmark P₃ corresponding to an end of the index finger in this order,by an iterative estimating step S153 according to the present invention.

FIG. 9 illustrates specific steps included in an iterative estimatingstep S153 according to the present invention.

FIG. 10 two-dimensionally illustrates a calculating step of Equation 6.

FIG. 11 illustrates an example which selects an appropriate root amongcandidates of a second joint of a middle finger of a hand of a body.

BEST MODE

The exemplary embodiment described in this specification may be entirelyhardware, or partially hardware and partially software or entirelysoftware. In this specification, “unit”, “module”, “device”, or “system”refers to a computer related entity, such as hardware, a combination ofhardware and software, or software. For example, in this specification,“unit”, “module”, “device”, or “system” may be a process which is beingexecuted, a processor, an object, an executable file, a thread ofexecution, a program, and/or a computer, but is not limited thereto. Forexample, both an application which is being executed in a computer andthe computer may correspond to “unit”, “module”, “device”, or “system”of this specification.

Exemplary embodiments are described with reference to flowchartssuggested in the drawings. In order to simply describe the presentinvention, the method has been illustrated and described as a series ofblock diagrams but the present invention is not limited to the order ofthe blocks and processes of some blocks may be performed in a differentorder from the order of the blocks illustrated and described in thespecification or simultaneously performed with other blocks. Further,various different divergences, flow paths, and orders of the blockswhich may achieve the same or similar results may be implemented.Further, all blocks which have been illustrated may not be required toimplement the method described in the specification. Moreover, a methodaccording to an exemplary embodiment of the present invention may beimplemented in the form of a computer program to execute a series ofprocesses and the computer program may be recorded in a computerreadable recording medium.

Hereinafter, a configuration and characteristics of the presentinvention will be described by way of exemplary embodiments, which arenot intended to be limiting, but merely illustrative of the invention.

Hereinafter, an articulated model registration apparatus and anarticulated model registration method will be described with referenceto the drawings.

FIG. 1 illustrates an exemplary embodiment of an articulated modelregistration apparatus 10 according to the present invention. Thearticulated model registration apparatus 10 includes a body imagedetecting unit 102 which detects an image of a body area from anoriginal image, a landmark extracting unit 103 which extracts one ormore landmarks including an articulated point from the detected image ofthe body area, a body shape model generating unit 104 which generates anaverage three-dimensional body shape model, a conversion estimating unit105 which estimates a position on a three-dimensional coordinate of thegenerated body shape model from one or more landmarks other than thearticulated point among the landmarks, and a posture estimating unit 106which estimates a position on a three-dimensional coordinate of thegenerated body shape model from one or more articulated points of thelandmarks.

The articulated model registration apparatus 10 according to anexemplary embodiment of the present invention may further include animage input unit 101. The image input unit 101 according to an exemplaryembodiment of the present invention may receive an original image in theform of electronic data. The image input unit 101 may be a camera 204 orreceive an original image input from the camera 204. Further, theoriginal image input unit 101 may receive the original image fromdatabase in which the original image is stored as electronic data.Further, the image input unit 101 may receive the original image from anexternal network connected to database in which the original image isstored as electronic data.

Further, the image input unit 101 may receive a camera matrixrepresenting a photographing referential point and direction on athree-dimensional coordinate of the original image.

The body image detecting unit 102 according to an exemplary embodimentof the present invention may detect an image of a body area from theoriginal image. As long as the body image detecting unit 102 detects animage of a body area from the original image, the body image detectingunit 102 is not limited and may be an object detector or use the objectdetector. The object detector may be a machine learning based detectorand for example, may be a single shot multibox detector (SSD) or youonly look once (YOLO).

The landmark extracting unit 103 according to an exemplary embodiment ofthe present invention, for example, detects a boundary vector betweenthe body area and a background area and then detects a body end pointand a point between body ends at the boundary vector. Specifically, inthe case of a hand area, a thumb area and an area of fingers other thanthe thumb may be detected. Further, during a process of detecting aboundary vector between the body area and the background area, forexample, an area in which a color value of each pixel of the detectedimage of the body area corresponds to a range of a skin color value andan area in which the color value does not correspond to a range of theskin color value are converted into a white area and a black area,respectively, and then the boundary line vector between the body areaand the background area may be detected.

Further, specifically, during a process of detecting a body end pointand a point between body ends at the boundary vector, for example, apoint corresponding to an inflection point which is equal to or lowerthan a predetermined angle is detected as a body end point and a pointbetween body ends. For example, the point may be detected by a deeplearning based method, specifically, a convolutional pose machine (CPM)based segmentation technique. Further, the boundary vector between thedetected body area and the background area is input to a Harris cornerdetector and when a neighbor vector including a point corresponding toan inflection point has a concave shape, the point is detected as a bodyend point and when the neighbor vector has a convex shape, the point isdetected as an inter-body point.

A three-dimensional body shape model generating unit 104 according to anexemplary embodiment of the present invention may load athree-dimensional body shape model from a three-dimensional body shapemodel database in which one or more three-dimensional body shape modelsare stored to generate the three-dimensional body shape model.

In the present invention, the “three-dimensional body shape model” is aposition information set on a three-dimensional space of an end pointand articulated points of the body and may refer to a three-dimensionalbody posture to be recognized from the original image. Specifically, thebody shape model may be a basis vector set of a body shape dispersionconstructed by the average three-dimensional body skeleton and principalcomponent analysis. The average three-dimensional body shape model maybe information on a shape and a size of a normal body with an averagesize of a body shape and a size used at the time of machine learning,for the purpose of registration of a three-dimensional body shape.

The conversion estimating unit 105 according to the exemplary embodimentof the present invention may estimate a position on a three-dimensionalcoordinate of the generated body shape model, not from an articulatedpoint, but from one or more landmarks among a camera matrix on thethree-dimensional coordinate and the landmarks of the original image,for example, by calling SolvePnP function of OpenCV.

Further, the posture estimating unit 106 may estimate the position onthe three-dimensional coordinate of the articulated point with respectto a straight line connecting the position on the three-dimensionalcoordinate estimated from any one or more of the articulated points, inan arbitrary position on the three-dimensional coordinate.

In the present invention, the “posture” may refer to a positioninformation set on the three-dimensional space of the end point andarticulated points of the body.

FIG. 2 illustrates an exemplary embodiment of an articulated modelregistration apparatus 10 according to the present invention. Thearticulated model registration apparatus 10 may include a memory 201, aprocessor 202, and a communication unit 203. The memory 201 may includea body image detecting unit 101, the landmark extracting unit 102, thebody shape model generating unit 103, the conversion estimating unit104, and the posture estimating unit 105. Specifically, the memory 201may be executed through the processor 202 by statically or dynamicallyallocating the corresponding unit/module.

The memory 201 may include an arbitrary combination of volatile andnonvolatile memory appropriate for an intended purpose (which may bedistributed or localized as appropriate) and for the convenience ofdescription, may include other memory segments which is not illustratedin this example. For example, the memory 208 may include a code storagearea, a code execution area, and a data area without departing from thescope of the present invention.

The communication unit 203 may provide a data communication function tosearch still image contents, audio and video contents or other contentsand other activities through a satellite, a cable, a storage medium,Internet, or other content providers if it is appropriate for a givenembodied example, and for example, may include a wireless communicationstandard such as wired Ethernet, cellular wireless communication, andBluetooth®.

FIG. 3 illustrates an exemplary embodiment of an articulated modelregistration apparatus 10 according to the present invention. Thearticulated model registration apparatus 10 may include a memory 201, aprocessor 202, a communication unit 203, a camera 204, and a display205. The memory 201 may include the body image detecting unit 101, thelandmark extracting unit 102, the body shape model generating unit 103,the conversion estimating unit 104, and the posture estimating unit 105.

The camera 204 may convert optical information into two-dimensionalpixel color information. Further, the camera 204 may converttwo-dimensional pixel depth information. Furthermore, the camera 204 maygenerate a camera matrix representing a photographing reference pointand direction on a three-dimensional coordinate of the original image.

In the present invention, the “image” may be data of two-dimensionalvisual information or data including depth information in addition tothe two-dimensional visual information. Further, the image maycorrespond to any one frame of a moving image formed of a plurality offrames.

As long as the display 205 optically displays images, the display 205 isnot limited and for example, may be a cathode ray tube, a liquid crystaldisplay device, a light emitting diode device, a plasma display, andfurther include a touch panel.

The articulated model registration apparatus 10 according to the presentinvention may perform an articulated model registration method whichwill be described below and all the description of the articulated modelregistration method described below may be applied.

FIG. 4 illustrates an exemplary embodiment of an articulated modelregistration method according to the present invention. The articulatedmodel registration method according to the present invention includes abody image detecting step S11 of detecting an image of a body area froman original image, a landmark extracting step S12 of extracting one ormore landmarks including an articulated point from the detected image ofthe body area, a body shape model generating step S13 of generating anaverage three-dimensional body shape model, a conversion estimating stepS14 of estimating a position on a three-dimensional coordinate of thegenerated body shape model from a conversion estimating landmarkincluding any one or more landmarks among the extracted landmarks, and aposture estimating step S15 of estimating a position on athree-dimensional coordinate of the generated body shape model from anyone or more landmarks among the extracted landmarks.

Further, the posture estimating step S15 may estimate the position onthe three-dimensional coordinate of the landmark from a referencestraight line connecting the position on the three-dimensionalcoordinate estimated from any one or more of the landmarks, in anarbitrary position on the three-dimensional coordinate.

The body posture recognition method according to an exemplary embodimentof the present invention may further include a step of receiving anoriginal image (not illustrated). Specifically, in the step of receivingan original image, the original image may be input in the form ofelectronic data, for example, input in the form of electronic data fromthe camera or the original image may be transmitted from the database inwhich the original image is stored as electronic data. Further, theoriginal image input unit 101 may receive the original image from anexternal network connected to a database in which the original image isstored as electronic data or receive the original image from the camera204.

The body image detecting step S11 according to an exemplary embodimentof the present invention may detect an image of a hand area from theoriginal image.

As long as the body image detecting step S11 detects an image of a bodyarea from the original image, the body image detecting step S11 is notlimited and may be an object detector or use the object detector. Theobject detector may be a machine learning based detector and forexample, may be a single shot multibox detector (SSD) or you only lookonce (YOLO).

The landmark extracting step S12 according to an exemplary embodiment ofthe present invention, for example, detects a boundary vector betweenthe body area and a background area and then detects a body end pointand a point between body ends at the boundary vector. Specifically, inthe case of a hand area, a thumb area and an area of fingers other thanthe thumb may be detected. Further, during a process of detecting aboundary vector between the body area and the background area, forexample, an area in which a color value of each pixel of the detectedimage of the body area corresponds to a range of a skin color value andan area in which the color value does not correspond to a range of theskin color value are converted into a white area and a black area,respectively, and then the boundary line vector between the body areaand the background area may be detected.

Further, specifically, during a process of detecting a point between abody end point and a body end at the boundary vector, for example, apoint corresponding to an inflection point which is equal to or lowerthan a predetermined angle is detected as a point between a body endpoint and a body end. For example, the point may be detected by a deeplearning based method, specifically, a convolutional pose machine (CPM)based segmentation technique. Further, the boundary vector between thedetected body area and the background area is input to a Harris cornerdetector and when a neighbor vector including a point corresponding toan inflection point has a concave shape, the point is detected as afinger end point and when the neighbor vector has a convex shape, thepoint is detected as a point between fingers.

The body shape model generating step S13 according to an exemplaryembodiment of the present invention may generate a three-dimensionalbody shape model by matching an average three-dimensional hand shapemodel from a three-dimensional body shape model database in which one ormore three-dimensional body shape models are stored.

The conversion estimating step S14 according to an exemplary embodimentof the present invention may estimate a position on a three-dimensionalcoordinate of the generated body shape model, not from an articulatedpoint, but one or more landmarks among a camera matrix on thethree-dimensional coordinate and the landmarks of the original image.For example, the conversion estimating step S14 may call SolvePnPfunction of OpenCV to estimate the position of the body shape model onthe three-dimensional coordinate.

In the conversion estimating step S14 according to an exemplaryembodiment of the present invention, the conversion estimating landmarkmay be a landmark corresponding to a joint which does not changeaccording to the change of the joint angle.

When a landmark for the hand is estimated, the conversion estimatinglandmark may be a landmark corresponding to any one selected from agroup consisting of a wrist, a root joint of an index finger, a rootjoint of a middle finger, a root joint of a ring finger, and a rootjoint of a small finger. A joint corresponding to a thumb and jointsother than roots of the remaining four fingers have a high degree offreedom by the pose of the hand as well as the position of the hand sothat the joints may not be used to estimate a position withoutestimating a pose which will be performed in a next step.

Further, when a landmark for the entire body is estimated, theconversion estimating landmark may be a landmark corresponding to ashoulder or hip joint.

FIG. 6 illustrates specific steps included in a posture estimating stepS15 according to the present invention.

The posture estimating step S15 according to the present invention mayinclude a step S151 of calculating a reference straight line connectinga camera matrix on a three-dimensional coordinate of the original imageand a landmark at which the position on the three-dimensional coordinateis estimated.

The posture estimating step S15 according to the present invention mayinclude a step S151′ of calculating a reference straight line with acamera matrix which is a projection reference matrix on athree-dimensional coordinate and a landmark on a two-dimensionalcoordinate projected by the camera matrix from the original image andthe process therefor is two-dimensionally illustrated in FIG. 5.

The step S151′ of calculating a reference straight line may include astep S151′ of calculating a reference straight line represented by thefollowing Equation 1.

P_(b)+D_(t)  [Equation 1]

In Equation 1, a direction vector D is represented by the followingEquation 2 and P_(b) is represented by the following Equation 3.

D=(C ₀ −C ₂ ·x″)×(C ₀ −C ₂ ·y″)  [Equation 2]

In Equation 2, C₀ and C₂ are first and third row components of thecamera matrix C which is a projection reference matrix on athree-dimensional coordinate from the original image and x″ and y″ arecoordinates on a two-dimensional coordinate system of the posturelandmark.

(C ₀ −C ₂ ·x″)·P _(b)=0,(C ₀ −C ₂ ·y″)·P _(b)=0  [Equation 3]

In Equation 3, P_(b) may be calculated by setting a z coordinate valueto be 0.

To be more specific, when there are a camera matrix C which is aprojection reference matrix on a three-dimensional coordinate and onevertex P_(3D) on the three-dimensional space, the 2D coordinate of thevertex is defined by the following Equation 3A.

(x′,y′,z′)=C·P _(3D) ,P _(2D)=(x″,y″)=(x′/z′,y′/z′)  [Equation 3A]

When P_(2D) and C are given, P_(3D) lies on a straight line. When it isassumed that the straight line is P_(b)+Dt, one point P_(b) and thedirection vector D of the straight line may be obtained by the followingmethod.

When individual rows of C are C₀, C₁, and C₂, D is calculated using thefollowing Equation 3B.

x′=C ₀ ·P _(3D) ,y=C ₁ ·P _(3D) ,z′=C ₂ ·P _(3D)

x″=C ₀ ·P _(3D) /C ₂ ·P _(3D) ,y″=C ₁ ·P _(3D) /C ₂ ·P _(3D)

(C ₀ −C ₂ ·x″)·P _(3D)=0,(C ₀ −C ₂ ·y″)·P _(3D)=0

D=(C ₀ −C ₂ ·x″)×(C ₀ −C ₂ ·y″)  [Equation 3B]

P_(b) obtains a cross point of the straight line and an xy plane. In theabove Equation 3, P_(b) is assumed as (x_(b), y_(b), 0) and x_(b) andy_(b) may be calculated by solving a simultaneous equation for twoexpressions and two unknowns.

The posture estimating step S15 according to the present invention mayinclude a step S152 of moving an estimation start landmark to a positionon a three-dimensional coordinate which is the most proximate from thereference straight line.

The posture estimating step S15 according to the present invention mayinclude an iterative estimating step S153 which estimates a position onthe three-dimensional coordinate of the landmark in the connecting orderfrom the estimation start landmark to an end landmark.

FIG. 7 illustrates an example which estimates positions on athree-dimensional coordinate of a landmark P₀ corresponding to ashoulder of the body, a landmark P₁ corresponding to an elbow, and alandmark P₂ corresponding to a wrist of the end of the body in thisorder, by an iterative estimating step S153 according to the presentinvention.

FIG. 8 illustrates an example which estimates positions on athree-dimensional coordinate of a landmark P₀ corresponding to a rootjoint of the hand of the body, specifically, the index finger, landmarksP₁ and P₂ corresponding to two joints of the index finger, and alandmark P₃ corresponding to an end of the index finger in this order,by an iterative estimating step S153 according to the present invention.

FIG. 9 illustrates specific steps included in an iterative estimatingstep S153 according to the present invention.

The iterative estimating step S153 according to the present inventionmay include a step S1531 of calculating a landmark root satisfying acondition that a distance to a straight line on a three-dimensionalcoordinate calculated by a landmark at which estimation is completed anda landmark which is a next connecting order is equal to the distancebetween the landmark at which estimation is completed and a landmarkwhich is a next connecting order, with respect to the landmark at whichestimation is completed.

As illustrated in FIG. 5, when a position of the existing joint is P_(c)and a distance between joints is r, a vertex which is located on athree-dimensional straight line corresponding to the landmark of thenext joint to be spaced apart from P_(c) by r may be obtained by solvingthe following Equation 4 with respect to t. Normally, there are tworoots of P_(a) and P_(b). However, there is one corresponding vertex(when the quadratic equation has multiple roots) or there is no vertex,depending on the distance r. Specifically, the iterative estimating stepS153 may include a step S1531′ of estimating a landmark which is a nextconnecting order by calculating a root of t by the following Equation 4,with respect to the landmark at which estimation is completed.

(P _(c) −P _(b) +D·t)² −r ²=0  [Equation 4]

In Equation 4, Pc is a position on the three-dimensional coordinate ofthe landmark at which estimation is completed, r is a distance between alandmark at which estimation is completed and a landmark which is a nextconnecting order, and P_(b) and D are equal to those of Equation 1.

When there is one landmark root, the iterative estimating step S153according to the present invention may include a step S1532 ofestimating a root of a landmark as a landmark which is a next connectingorder of a landmark at which estimation is completed.

A direction in which the body joint bends is limited. It is possible toinspect whether the body joint includes a candidate root in a normallybendable range and when there are two landmark roots, the iterativeestimating step S153 may include a step S1533 of estimating a rootincluded in a movable area of a body joint as a landmark which is a nextconnecting order of the landmark at which estimation is completed.

The step S1533 of estimating to be a landmark may include a step S1533′of estimating P_(candidate) satisfying the following Equation 5 by alandmark which is a next connecting order.

(P _(candidate) −P _(c))·N  [Equation 5]

In Equation 5, Pc is a position of a landmark at which estimation iscompleted, on the three-dimensional coordinate and N is a vector of anorm direction represented by the following Equation 6.

N=L×C  [Equation 6]

In Equation 6, L may be P_(candidate)−P_(c), that is, a link directionwhich is a difference between the position on the three-dimensionalcoordinate and a position on the three-dimensional coordinate of theroot joint, C may be a cross direction which is a difference of any twolandmarks among the estimation start landmarks. The above Equation 6will be two-dimensionally illustrated in FIG. 10.

For example, when P_(candidate) which is a candidate of a first joint isselected at the time of estimating a landmark of the hand, the firstjoint of each finger is bent in a palm direction, so that the palmdirection may be set as a norm direction.

When there is no landmark root, the iterative estimating step S153according to the present invention may include a step S1534 ofestimating a position which is the shortest distance from Pc in thereference straight line as a landmark which is a next connecting order.Specifically, in the reference straight line, P_(c) and the shortestdistance may be the midpoint of the straight line on a three-dimensionalcoordinate between P_(A) and P_(B).

FIG. 11 illustrates an example which selects an appropriate root amongcandidates of a second joint of a middle finger of a hand of a body. Cof the above Equation 6 may be a cross direction which is a differenceof the vector on the three-dimensional coordinate between a landmarkcorresponding to a root joint of a small finger and a landmarkcorresponding to a root joint of an index finger.

The present invention may provide a program including a command forexecuting the articulated model registration method.

The present invention may provide a computer readable recording mediumin which the above-described program is stored.

The specified matters and limited exemplary embodiments and drawingssuch as specific elements in the present invention have been disclosedfor broader understanding of the present invention, but the presentinvention is not limited to the exemplary embodiments, and variousmodifications, additions and substitutions are possible from thedisclosure by those skilled in the art.

The spirit of the present invention is defined by the appended claimsrather than by the description preceding them, and all changes andmodifications that fall within metes and bounds of the claims, orequivalents of such metes and bounds are therefore intended to beembraced by the range of the spirit of the present invention.

1. An articulated model registration method, comprising: a body imagedetecting step of detecting an image of a body area from an originalimage; a landmark extracting step of extracting one or more landmarksincluding an articulated point from the detected image of the body area;a body shape model generating step of generating an averagethree-dimensional body shape model; a conversion estimating step ofestimating a position on a three-dimensional coordinate of the generatedbody shape model from a conversion estimating landmark including any oneor more landmarks among the extracted landmarks; and a postureestimating step of estimating a position on a three-dimensionalcoordinate of the generated body shape model from any one or morelandmarks among the extracted landmarks, wherein the posture estimatingstep estimates the position on the three-dimensional coordinate of thelandmark from a reference straight line connecting the position on thethree-dimensional coordinate estimated from any one or more of thelandmarks, in an arbitrary position on the three-dimensional coordinate.2. The articulated model registration method of claim 1, wherein theposture estimating step includes a step of calculating a referencestraight line connecting a camera matrix on a three-dimensionalcoordinate of the original image and a landmark at which a position onthe three-dimensional coordinate is estimated.
 3. The articulated modelregistration method of claim 2, wherein the step of calculating thereference straight line includes a step of calculating the referencestraight line by the camera matrix which is a projection referencematrix on the three-dimensional coordinate from the original image andthe landmark on the two-dimensional coordinate projected by the cameramatrix.
 4. The articulated model registration method of claim 2, whereinthe posture estimating step includes a step of moving an estimationstart landmark to a position on the three-dimensional coordinate whichis the closest from the reference straight line.
 5. The articulatedmodel registration method of claim 4, wherein the posture estimatingstep includes a step of moving the estimation start landmark to aposition on the three-dimensional coordinate which is the closest fromthe reference straight line.
 6. The articulated model registrationmethod of claim 1, wherein the posture estimating step includes aniterative estimating step of estimating a position on athree-dimensional coordinate of the landmark in a connecting order fromthe estimation start landmark to an end landmark.
 7. The articulatedmodel registration method of claim 6, wherein the iterative estimatingstep includes a step of calculating a landmark root satisfying acondition in which a distance from the straight line on thethree-dimensional coordinate calculated by a landmark at whichestimation is completed and a landmark which is a next connecting orderis equal to the distance between the landmark at which estimation iscompleted and the landmark which is a next connecting order, withrespect to the landmark at which estimation is completed.
 8. Thearticulated model registration method of claim 7, wherein when there aretwo landmark roots, the iterative estimating step includes a step ofestimating a root included in a movable area of a body joint as alandmark which is a next connecting order of a landmark at whichestimation is completed.
 9. The articulated model registration method ofclaim 7, wherein when there is no landmark root, the iterativeestimating step includes a step of estimating a position which is theshortest distance from Pc in the reference straight as a landmark whichis a next connecting order.
 10. An articulated model registrationapparatus, comprising: a body image detecting unit which detects animage of a body area from an original image; a landmark extracting unitwhich extracts one or more landmarks including an articulated point fromthe detected image of the body area; a body shape model generating unitwhich generates an average three-dimensional body shape model; aconversion estimating unit which estimates a position on athree-dimensional coordinate of the generated body shape model from oneor more landmarks other than the articulated point among the landmarks;and a posture estimating unit which estimates a position on athree-dimensional coordinate of the generated body shape model from oneor more articulated points among the landmarks, wherein the postureestimating unit estimates the position on the three-dimensionalcoordinate of the landmark from a reference straight line connecting theposition on the three-dimensional coordinate estimated from any one ormore of the landmarks, in an arbitrary position on the three-dimensionalcoordinate.